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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON,  VIRGINIA  22202-4704 


February  21 ,  2001 

MEMORANDUM  FOR  DIRECTOR,  DEFENSE  LOGISTICS  AGENCY 

SUBJECT :  Audit  Report  on  the  Defense  Logistics  Agency  Product  Verification 
Program  (Report  No.  D-200 1-054) 


We  are  providing  this  report  for  your  information  and  use.  We  conducted  this 
audit  to  determine  how  items  purchased  and  stocked  by  the  Defense  Logistics  Agency 
were  selected  for  testing  and  how  the  test  results  were  used. 

The  Defense  Logistics  Agency  comments  conformed  to  the  requirements  of 
DoD  Directive  7650.3;  therefore,  additional  comments  are  not  required. 

Questions  on  the  audit  should  be  directed  to  Mr.  Nicholas  E.  Como  at 
(703)  604-9215  (DSN  664-9215)  (ncomo@dodig.osd.mil)  or  Mr.  Terry  L.  McKinney 
at  (703)  604-9288  (DSN  664-9288)  (tmckinney@dodig.osd.mil).  See  Appendix  D  for 
the  report  distribution.  The  audit  team  members  are  listed  inside  the  back  cover. 

David  K.  Steensma 
Deputy  Assistant  Inspector  General 
for  Auditing 
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Report  No.  D-2001-054 

(Project  No.  D2000CF-011 1.001) 

Defense  Logistics  Agency 
Product  Verification  Program 


February  21,  2001 


Executive  Summary 


Introduction.  This  is  the  second  of  two  reports  being  issued  by  the  Inspector  General, 
DoD,  which  addresses  various  aspects  of  the  Defense  Logistics  Agency  Product 
Verification  Program.  The  Product  Verification  Program  was  established  in  January 
1995  to  consolidate  test  program  management  and  activities  for  improved  efficiency, 
consistency,  and  reduced  operational  costs.  In  FY  1999,  the  Product  Verification 
Program  managers  reported  operational  costs  of  $6.6  million  and  tested  over 
10,000  items.  This  report  addresses  how  products  were  selected  for  testing  and  how 
the  test  results  were  used. 

Objectives.  Our  objective  was  to  evaluate  whether  the  Defense  Logistics  Agency  was 
effectively  managing  the  Product  Verification  Program.  Specifically,  the  audit 
determined  how  products  were  selected  for  testing  and  whether  the  program’s  testing 
plan  was  adequate.  The  audit  also  determined  whether  the  Product  Verification 
Program  managers  and  quality  assurance  specialists  were  using  the  test  results  to 
identify  contractor  problems  and  purge  potentially  defective  products  from  the  Defense 
Logistics  Agency  depots.  We  also  reviewed  the  management  control  program  as  it 
related  to  the  overall  audit  objective. 

Results.  The  Defense  Logistics  Agency  product  test  center  planning  procedures  were 
logical  and  in  conformance  with  test  objectives.  Testing  was  conducted  using  contract 
specifications  and  objectives,  appropriate  test  equipment  was  used,  and  suspected 
deficiencies  were  evaluated.  However,  the  product  test  selections  and  the  use  of  test 
results  needed  improvement.  Random  product  test  selections  did  not  include  all 
products  available  for  testing  at  all  depots.  For  nonrandom  testing,  the  Product 
Verification  Office  did  not  fully  consider  management's  quality  priorities  and  initiatives 
in  test  planning.  As  a  result,  funds  for  product  testing  were  not  used  in  the  most 
efficient  manner  and  DoD  lacked  sufficient  assurance  that  some  critical  products  would 
perform  as  expected  (finding  A).  For  two  of  the  three  Defense  Supply  Centers,  test 
failures  were  not  consistently  investigated  and  required  actions  on  test  failures  were  not 
always  taken.  Inconsistent  adjudication  and  ratings  of  test  results  hindered  the  two 
Defense  Supply  Centers  from  resolving  contractor  issues  for  36  percent  of  the 
231  FY  1999  tests  we  reviewed,  inflated  quality  ratings  for  as  many  as  54  contractors 
and  allowed  potentially  nonconforming  products  to  remain  available  for  issue 
(finding  B) .  See  Appendix  A  for  details  on  the  management  control  program. 
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Summary  of  Recommendations.  We  recommend  that  the  Director,  Defense  Logistics 
Agency,  establish  random  testing  selection  methods,  and  implement  nonrandom  testing 
initiatives.  We  also  recommend  that  the  Director,  Defense  Logistics  Agency,  develop 
and  implement  uniform  training  for  quality  assurance  specialists  and  supervisory  review 
procedures.  We  also  recommend  documenting  causative  factors  of  test  failures  and 
determining  whether  nonconforming  products  should  be  suspended  or  reevaluated. 

Management  Comments.  The  Defense  Logistics  Agency  stated  that  it  is  developing  a 
new  random  testing  selection  program  by  Fiscal  Year  2002  that  will  include  the 
recommended  criteria  for  selection.  The  Defense  Logistics  Agency  agreed  to 
implement  nonrandom  testing  initiatives  prescribed  in  its  agency  guidance  and  agreed  to 
perform  trend  analyses  from  quality  deficiency  reports.  The  Defense  Logistics  Agency 
also  agreed  to  enhance  existing  training  and  supervisory  review  programs  for  Quality 
Assurance  Specialists  and  has  begun  to  reevaluate  and  take  appropriate  corrective  action 
on  the  test  results  disclosing  nonconforming  products  identified  in  this  report.  See  the 
Finding  section  of  the  report  for  a  discussion  of  management  comments  and  to  the 
Management  Comments  section  of  the  report  for  the  complete  text  of  the  comments. 
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Background 

Product  Verification  Program.  The  Product  Verification  Program  (PVP)  was 
created  in  January  1995  by  Defense  Logistics  Agency  (DLA)  Directive  4105.20. 
The  Product  Verification  Program  replaced  the  Contractor  Assessment  Product 
Evaluation  Program.  The  program  establishes  the  policies  and  responsibilities 
for  laboratory  testing  and  product  verification  and  includes  all  testing  performed 
within  the  supply  centers.  The  primary  objectives  of  PVP  are  to  improve  the 
process  of  delivering  logistical  support  at  reduced  costs  by: 

•  providing  quality  material  that  meets  DLA  customer  requirements, 

•  reducing  failure  costs  incurred  for  nonconforming  material  received 
from  suppliers  by  verifying  conformance  of  material  to  contract 
specifications,  and 

•  utilizing  supplier’s  past  performance  for  future  source  selections  and 
“best  value”  contracting  decisions. 

In  FY  1999,  22  personnel  were  assigned  to  the  PVP  at  the  Defense  Supply 
Centers.  Over  4,700  products  were  randomly  selected  for  testing  and  about 
5,600  products  were  tested  on  a  nonrandom  basis  during  FY  1999.  These 
10,300  product  tests  resulted  in  nearly  1,200  laboratory  test  failures.  The 
FY  1999  operating  budget  for  the  PVP  was  $6.6  million.  The  supply  centers 
reported  critical  and  major  defects  between  FYs  1998  to  2000  ranging  between 
12  and  18  percent  of  the  DLA-managed  products  tested. 

Objectives 

Our  objective  was  to  evaluate  whether  the  Defense  Logistics  Agency  was 
effectively  managing  the  PVP.  Specifically,  the  audit  determined  how  products 
were  selected  for  testing  and  whether  the  program’s  testing  plan  was  adequate. 
The  audit  also  determined  whether  PVP  managers  and  quality  assurance 
specialists  were  using  test  results  to  identify  contractor  problems  and  purge 
potentially  defective  products  from  the  DLA  depots.  We  also  reviewed  the 
management  control  program  as  it  related  to  the  overall  audit  objective.  See 
Appendix  A  for  a  discussion  of  the  audit  process  and  the  management  control 
program  review. 
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A.  Product  Selection  for  Quality  Testing 

The  Product  Verification  Program  office  had  not  established  a  uniform 
process  to  select  DLA  managed  products  for  quality  testing.  The  process 
was  ineffective  because  all  available  products  were  not  included  for 
random  testing  and  product  criticality  was  not  considered.  Further, 
management  initiatives  were  not  fully  implemented  to  select  products  for 
testing  based  on  factors  such  as  high  requisition  rates,  high  cost,  or 
high-potential  failure  rates  for  nonrandom  testing.  As  a  result,  testing 
resources  were  not  expended  efficiently. 


Random  Selection  Criteria  and  Application 

DLA  Directive  4105.20,  Product  Verification  Program  for  Inventory  Control 
Points,  January  20  1995,  directs  PVP  offices  at  Defense  Supply  Centers  to 
perform  random  laboratory  testing  of  the  following  categories  of  incoming 
material. 

•  Material  identified  on  new  contracts  less  than  3  years  old. 

•  In-stock  material  identified  on  contracts  older  than  3  years  old. 

•  Material  with  the  highest  failure  costs  identified  on  weapon  and  safety 
critical  parts. 


Sampling  Assistance  Model.  DLA  employed  the  “Sampling  Assistance 
Model,”  to  generate  a  statistically  valid  selection  of  DLA  managed  products  for 
quality  testing.  The  model  was  developed  and  maintained  by  the  Operations 
Research  and  Resource  Analysis  office.  This  office  generated  a  quarterly  listing 
of  randomly  selected  products  by  national  stock  number  for  each  Defense 
Supply  Center.  The  PVP  offices  at  the  Defense  Supply  Centers  used 
information  from  the  quarterly  listings  to  select  inventory  products  for  quality 
testing. 

Products  Subjected  to  Random  Testing 


The  random  product  selections  did  not  include  all  products  that  were  stocked  at 
all  the  depots.  The  products  subjected  to  random  testing  were  limited  to 
products  stocked  at  the  two  primary  distribution  sites,  San  Joaquin,  California, 
and  New  Cumberland,  Pennsylvania,  and  the  Richmond  Distribution  Center. 
Therefore,  randomly  selected  products  were  not  included  from  the  remaining 
21  DLA  depots  (the  nonprimary  distribution  sites).  During  June  2000,  the 
Operations  Research  and  Resources  Analysis  office  provided  information  on  the 
location  of  DLA  managed  products.  Table  1  shows  the  location  and  percentages 
of  DLA  products  that  were  part  of  the  random  sample  universe. 
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Table  1.  Location  of  DLA  Products 

(in  millions) 


Total 

Stocked  at 

Percent 
Stocked  at 

Stocked  at 

Percent 
Stocked  at 

Products 

Primary 

Primary 

Nonprimary 

Nonprimary 

Stocked  at  all 

Distribution 

Distribution 

Distribution 

Distribution 

DLA  Depots 

Sites 

Sites 

Sites 

Sites 

Columbus 

1,175 

677 

58 

498 

42 

Richmond 

617 

253 

41 

364 

59 

Philadelphia 

659 

326 

49 

333 

51 

Total 

2,451 

1,256 

51 

1,195 

49 

A  total  of  nearly  1.2  million  products,  or  49  percent  of  the  DLA  stocked 
products,  were  located  in  the  nonprimary  distribution  sites  and  were  not 
included  in  the  random  sample  universe.  Examples  of  products  stocked  in 
nonprimary  distribution  sites  that  failed  nonrandom  testing  but  were  never  tested 
on  a  random  basis  were: 

•  a  mirror  head  and  support  valve  assemblies  used  on  the  heavy  expanded 
mobility  tactical  truck,  and 

•  a  gear  shaft  that  was  housed  on  the  F-18  fighter  jet. 

These  critical  products  should  have  been  included  in  the  universe  for  random 
testing. 


Random  Selection  Categories 


For  FY  1999,  the  sampling  practices  of  the  PVP  offices  at  the  supply  centers 
were  not  in  full  compliance  with  the  random  selection  categories  outlined  in 
DLA  Directive  4105.20.  The  random  sample  universe  was  used  to  select  from 
both  a  total  and  limited  universe  of  DLA  products  scheduled  for  delivery  within 
6  months  of  the  fiscal  quarter.  Table  2  displays  the  categories  and  number  of 
random  tests  conducted  by  each  supply  center  during  FY  1999. 
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Table  2.  Random  Tests  Performed  by  Category  -  FY  1999 

Columbus 

Philadelphia 

Richmond 

Overall  inventory  (contracts  >  3  years) 

866 

420 

651 

New  receipts  (contracts  <  3  years)* 

744 

296 

137 

Weapon  system  critical  components 

- 

- 

806 

Customer  item  transfers 

785 

- 

- 

Total 

2,395 

716 

1,594 

*Limited  to  contracts  with  scheduled  deliveries  within  6  months. 

Only  the  PVP  office  at  the  Richmond  supply  center  randomly  tested  weapon 
system  critical  components.  The  components  included  structural  aircraft 
components,  helicopter  rotor  blades,  jet  turbine  engines,  and  aircraft  fuel  system 
components.  The  PVP  offices  at  the  Philadelphia  and  Columbus  supply  centers 
did  not  randomly  test  inventory  specifically  identified  as  critical. 

Random  Tests  of  Critical  Items.  We  further  analyzed  selected  products  to 
determine  criticality  within  the  random  selection  categories.  The  supply  centers 
did  not  consider  product  criticality  when  DLA  managed  products  were  randomly 
selected  for  quality  testing.  Table  3  shows  that  only  44  to  51  percent  of  the 
randomly  tested  products  were  listed  in  the  DLA  database  as  critical  products. 


Table  3.  Criticality  of  Randomly  Tested  Products  -  FY  1999 


Products 

Tested 

Critical 

Products 

Percent  of 

Non¬ 

Percent  of 

Critical 

critical 

Noncritical 

Products 

Products 

Products 

Columbus 

900* 

400 

44 

500 

56 

Richmond 

1,594 

806 

51 

788 

49 

Philadelphia 

716 

363 

51 

353 

49 

Total 

3,210 

1,569 

1,641 

*Review  limited  to  900  of  the  2,398  random  products  tested. 


The  DLA  guidance  specifically  addresses  product  criticality  for  random  product 
testing.  Tests  were  required  for  items  with  the  highest  potential  failure  rate. 
Critical  products  included  all  products  identified  as  weapon  system  critical, 
safety  critical,  or  all  other  parts  identified  as  critical  applications. 

Cost  of  Noncritical  Items.  We  further  analyzed  the  unit  cost  of  noncritical 
items  that  were  selected  for  random  testing  in  FY  1999.  Unit  cost  information 
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was  available  for  only  1,063  of  the  1,641  (65  percent)  noncritical  items 
randomly  selected  for  testing.  Table  4  displays  the  unit  cost  range  for  the 
1,063  noncritical  items  randomly  selected  for  testing. 


Table  4.  Unit  Cost  of  Randomly  Selected 
Noncritical  Items  for  Testing  FY  1999 


Unit  Cost  Range 

Number  of  Items 

Percent 

No  unit  cost  data  available 

578 

35 

Less  than  $2 

161 

10 

$2  to  $10 

300 

18 

$10  to  $50 

290 

18 

$  50  to  $100 

114 

7 

$  100  to  $  500 

140 

9 

$  500  to  $1,000 

32 

2 

$1,000  to  $2,000 

24 

1 

Greater  than  $2,000 

2 

- 

Total 

1,641 

100 

Over  50  percent  of  the  noncritical  items  that  were  randomly  selected  for  testing 
had  unit  costs  of  less  than  $100.  Selecting  low-cost,  noncritical  items  for  testing 
further  compromised  efficient  expenditures  of  testing  resources. 

Direct  Vendor  Deliveries.  The  PVP  offices  had  no  program  in  place  for  testing 
products  procured  through  the  Direct  Vendor  Delivery  Program.  These 
products  were  not  stored  in  the  depot  system  for  DoD  customer  use.  Instead, 
the  vendors  participating  in  this  program  shipped  products  directly  to  DoD 
customers.  As  of  October  1999,  56  contractors  were  providing  direct  vendor 
delivery  to  DLA.  This  form  of  supply  management  is  seemingly  the  way  future 
deliveries  will  be  completed.  Accordingly,  the  PVP  offices  must  address  how 
these  products  will  be  incorporated  into  the  Direct  Vendor  Delivery  Program. 


Nonrandom  Selection  Criteria 

The  product  verification  program  manager  must  develop  and  implement 
nonrandom  testing  initiatives  to  identify,  preclude  acceptance  of,  or  remove 
high-potential-for- failure  products  from  inventory.  The  initiatives  should  be 
based  on  customer  or  internal  DLA  management  feedback,  complaints,  or 
random  test  results.  Nonrandom  tests  were  also  directed  for  the  following  types 
of  products: 

•  with  known  or  potential  problems, 

•  procured  from  high-dollar  value  or  high- volume  contracts, 
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•  products  that  were  highly  requisitioned,  and 

•  procurements  from  contractors  with  poor  past  performance  history. 

DLA  Instruction  4155.2,  “Quality  Assurance  Program  Instruction  for  DLA,” 
February  17,  1999,  further  suggests  that  the  supply  centers  perform  trend 
analyses  of  products  found  to  be  deficient  on  a  recurring  basis. 

Table  5  displays  the  number  of  nonrandom  testing  conducted  by  the  three  PVP 
offices  during  FY  1999. 


Table  5.  Nonrandom  Tests  Performed  -  FY  1999 

Number  of  Nonrandom-Tests 

Columbus 

2,807* 

Richmond 

407 

Philadelphia  (general  and  industrial) 

341 

Philadelphia  (clothing  and  textiles) 

2,032 

Total 

5,587 

*  The  Electronics  Laboratory,  prior  to  reorganization,  conducted  1,675  of  the  2,807  tests. 

Nonrandom  Selection  Procedures 


Supply  center  PVP  offices  performed  nonrandom  tests  on  specific  products,  or 
groups  of  products,  when  Military  Departments  or  quality  assurance  specialists 
at  the  supply  centers  requested  assistance.  The  Defense  Supply  Center 
Philadelphia’s  Clothing  and  Textiles  Division  also  nonrandomly  tested  cloth 
samples  prior  to  fabrication  of  clothing  apparel.  The  PVP  offices  conducted 
nonrandom  testing  in  support  of  the  Counterfeit  Material/Unauthorized  Product 
Substitution  Program.  The  offices  also  performed  contract  mandated  testing  on 
preaccepted  products  from  a  contractor,  the  first  article  produced  by  a 
contractor,  and  portions  of  the  contractor’s  production. 

Uniformity  of  Nonrandom  Selection  Procedures.  DLA  Directive  4105.20 
establishes  guidelines  for  a  nonrandom  product  verification  selection  process. 
The  guidelines  ensure  that  the  most  costly,  critical,  and  safety-related  products 
having  a  history  of  suspicious  quality,  are  constantly  subjected  to  quality  review 
and  possible  verification  testing.  The  following  paragraphs  summarize  the 
nonrandom  product  selection  processes  implemented  by  each  supply  center 
during  FY  1999. 

•  Defense  Supply  Center,  Columbus.  The  Columbus  Electronics 
Division  tested  electronics  products  from  contractors  with  histories 
of  known  problems.  This  testing  was  discontinued  after  the 
Electronics  Test  Laboratory  was  reorganized  and  relocated  from 
Dayton,  Ohio,  to  Columbus,  Ohio.  The  Electronics  Test 
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Laboratory  conducted  1,675  tests  prior  to  termination.  During 
FY  1999,  the  Columbus  Center  initiated  a  “test  for  cause”  program 
on  new  contract  awards.  This  program  compared  new  contract 
awards  against  contractors  with  poor  performance  records.  The  test 
for  cause  program  was  used  exclusively  at  the  Columbus  Center. 

•  Defense  Supply  Center,  Richmond.  The  Richmond  Center 
established  the  “product  receipt-inventory  control  point/ depot 
evaluation  pilot”  program.  This  program  identified  newly  acquired 
products  that  were  critical,  highly  requisitioned,  and  had  a  history 
of  past  deficiencies.  The  program  was  terminated  during  FY  1999 
after  100  tests  resulted  in  two  test  failures.  The  Richmond  Center 
planned  to  initiate  a  flight  safety  critical  program  in  FY  2000. 

•  Defense  Supply  Center,  Philadelphia.  The  Philadelphia  Center 
had  not  established  a  nonrandom  process  to  identify  critical,  highly 
requisitioned  products  with  a  history  of  past  deficiencies. 
Nonrandom  tests  were  performed  to  create  new  diagrams,  eliminate 
duplicate  products,  and  support  Army  and  Air  Force  quality  issues 
on  critical  products. 

The  supply  center  PVP  offices  had  not  consistently  implemented  standard  DLA 
quality  initiatives  or  reported  results  of  trend  analyses  of  product  tests  that 
identified  product  quality  concerns  and  provided  the  basis  for  nonrandom 
testing.  Aside  from  the  Counterfeit  Material/Unauthorized  Product  Substitution 
program  tests,  testing  mandated  by  contract  or  requested  testing,  the  nonrandom 
selection  process  did  not  fully  comply  with  the  DLA  directive. 

Trend  Analyses  for  Recurring  Deficiencies 


Customers  that  requisitioned  DLA  managed  products  reported  instances  of  poor 
quality  or  nonconformity  on  a  Product  Quality  Deficiency  Report.  The  report 
provided  DoD  managers  with  a  valuable  tool  that  quickly  identified  product  and 
contractor  deficiencies  before  the  logistical  pipeline  was  overburdened  with 
nonconforming  items.  Prior  IG,  DoD,  audit  reports  (and  Finding  B  of  this 
report)  indicated  that  DLA  had  not  fully  obtained  the  potential  benefits  from  the 
Product  Quality  Deficiency  Report  program.  The  quality  assurance  program, 
when  effectively  implemented,  will  provide  an  invaluable  management  oversight 
mechanism  for  DoD  logistics  management. 

FY  1999  Deficiency  Reports.  During  FY  1999,  DLA  customers  forwarded 
over  8,000  quality  deficiency  reports  to  the  three  supply  centers’  for  review  and 
resolution.  Quality  assurance  specialists  were  required  to  take  corrective  action 
on  individual  deficiency  reports.  However,  during  FY  1999,  the  supply  centers 
did  not  perform  any  trend  analyses  of  recurring  deficiencies. 

Analysis  of  Deficiency  Reports.  We  reviewed  FY  1999  quality  deficiency 
reports  for  DLA  managed  products  that  were  defined  as  critical,  requisitioned 
more  than  15  times,  and  with  unit  values  in  excess  of  $250.  We  also  reviewed 
historical  deficiency  reports.  Overall,  we  reviewed  442  deficiency  reports 
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corresponding  to  245  DLA  managed  products  and  found  that  37  products  either 
warranted  testing  or  highlighted  a  need  for  contractor  surveillance.  In  addition, 
seven  other  products  required  updated  technical  data  or  revised  diagrams.  The 
following  paragraphs  are  examples  of  recurring  deficiencies  where  product 
testing  was  not  proposed  and  corrective  action  was  not  taken. 

As  of  July  2000,  several  complaints  were  received  regarding  an  AH-64  Apache 
helicopter  power  control  lever.  We  questioned  whether  flight  control 
mechanism  testing  was  required  to  determine  the  nature  of  the  problem.  After 
conferring  with  the  quality  assurance  specialists,  the  specialist  agreed  that 
testing  would  have  determined  the  nature  of  the  reported  deficiency. 

As  of  July  2000,  10  deficiency  reports  were  submitted  on  an  electrical  lead 
conduit  device.  The  customers  considered  the  conduit  unserviceable.  We  spoke 
to  PVP  office  personnel  and  they  concurred  that  verification  testing  would  have 
shown  contractor  noncompliance.  A  trend  analysis  will  provide  beneficial 
insight  on  those  products  that  warrant  testing  and  corrective  action. 

Recommendations,  Management  Comments  and  Audit 
Response 

A.  We  recommend  that  the  Director,  Defense  Logistics  Agency: 

1.  Establish  random  testing  selection  methods  in  Defense  Supply 
Center  product  verification  programs  to: 

a.  Sample  all  stored  depot  products. 

b.  Factor  product  criticality. 

c.  Include  direct  vendor  delivery  procurements. 

Management  Comments.  The  Defense  Logistics  Agency  nonconcurred  and 
stated  that  it  is  currently  developing  a  new  program  to  randomly  select  products 
for  testing.  The  program  uses  a  total  quality  cost  methodology  and  will  consider 
product  selection  criteria  based  on  depot  location,  product  criticality,  delivery 
methods,  and  customer  complaint  trends.  This  program  is  scheduled  to  replace 
the  current  random  selection  method  in  FY  2002. 

Audit  Response.  Although  the  Defense  Logistics  Agency  nonconcurred  with 
the  recommendation,  actions  proposed  to  improve  the  random  selection  of 
products  for  testing  satisfies  the  intent  of  the  recommendation.  No  further 
comments  are  required. 
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2.  Implement  nonrandom  testing  initiatives  prescribed  in  Defense 
Logistics  Agency  Directive  4105.20. 

3.  Prepare  trend  analyses  from  quality  deficiency  reports  to  identify 
nonconforming  inventory  for  the  following  products: 

a.  Critical. 

b.  Highly  requisitioned. 

c.  High-cost,  high-dollar,  or  high-volume. 

d.  Items  with  known  or  potential  deficiencies. 

e.  Items  from  contractors  with  poor  performance  histories. 

Management  Comments.  The  Defense  Logistics  Agency  concurred  and  will 
implement  nonrandom  testing  initiatives  prescribed  in  the  Defense  Logistics 
Agency  Directive  and  will  focus  on  testing  particular  items,  locations,  and 
suppliers  based  on  potential  risk  and  past  performance.  Also,  the  Defense 
Logistics  Agency  will  direct  its  supply  centers  to  implement  existing  Defense 
Logistics  Agency  policy  to  perform  trend  analyses  from  quality  deficiency 
reports.  The  agency  added  that  the  program  being  developed  will  randomly 
select  products  for  testing  and  also  will  identify  trends  of  repetitive  product 
deficiencies. 


9 


B.  Use  of  Test  Results 

Test  failures  were  not  always  investigated  and  subsequent  corrective 
actions  were  not  always  completed  on  tested  products.  Two  supply 
centers  inconsistently  rated  finalized  test  results  and  the  other  center  lost 
accountability  for  22  failed  test  results  from  an  automated  database. 
These  test  failures  and  lost  results  occurred  because  quality  assurance 
specialists  needed  additional  training  and  supervision.  As  a  result, 
inconsistent  ratings  and  adjudication  of  test  results  hindered  the  centers’ 
efforts  to  resolve  contractor  issues  of  quality  for  83  of  231  (36  percent) 
FY  1999  finalized  product  tests,  inflated  quality  ratings  for  54  DLA 
contractors,  and  made  potentially  nonconforming  products  available  for 
issue. 

Test  Results  Criteria 


Quality  Assurance  Policy.  Defense  Logistics  Agency  Directive  4155.2, 
“Quality  Assurance  Program  for  the  Defense  Logistics  Agency  Inventory 
Control  Points,”  October  10,  1997,  prescribes  that  reports  of  deficient  material 
should  be  adequately  and  completely  investigated  and  properly  documented. 
Quality  assurance  personnel  were  required  to  determine  the  necessary  corrective 
action,  address  the  cause  of  the  deficiency,  and  reflect  results  of  contractor- 
caused  deficiencies  in  the  contractor  quality  scoring  system  database. 

Quality  Assurance  Responsibility.  Defense  Logistics  Agency 
Instruction  4155.2  “Quality  Assurance  Instruction  for  DLA  ICP’s”  (Supply 
Centers),  February  17,  1999,  discusses  the  use  of  the  cause  code,  “simplified 
investigation,”  for  noncritical  items,  test  results  disclosing  minor  defects,  or 
product  tests  of  low  dollar  value.  However,  the  quality  assurance  specialist  was 
directed  to  assign  the  “contractor  noncompliance”  cause  code  when  the 
specialist  was  convinced  that  the  contractor  caused  the  defect.  The  instruction 
further  addresses  corrective  actions  when  quality  testing  reveals  defective 
products.  Corrective  actions  that  addressed  existing  deficiencies  included: 
issuing  alert  notifications  to  supply  screening  points  and  users,  inspecting 
existing  inventories,  and  recommending  contractual  warranty  enforcement  issues 
to  the  contracting  officer.  Other  examples  of  corrective  actions  that  precluded 
product  deficiency  reoccurrences  were:  recommending  specification  or  drawing 
changes  to  the  engineering  support  organization,  if  applicable;  modifying  the 
technical  data  file,  if  necessary;  and  providing  deficient  product  information  to 
contracting  and  logistics  offices. 

Investigating  and  Scoring  Test  Results 


In  FY  1999,  DLA  reported  that  nearly  10,000  products  were  quality  tested  and 
nearly  1,200  products  failed  laboratory  testing.  The  test  failures  ranged  from 
minor  defects  that  would  not  impair  product  use  or  operation,  to  major  defects 
that  would  leave  the  products  nonoperational.  The  PVP  records  disclosed  that 
671  deficiency  reports  were  finalized  in  FY  1999.  We  screened  the  deficiencies 
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for  671  finalized  FY  1999  Product  Quality  Deficiency  Reports.  The  screenings 
only  included  the  deficiencies  that  addressed  quality  issues,  resulted  in  a 
reported  cause  code  of  contractor  noncompliance,  or  a  simplified  investigation 
or  no  investigation.  We  further  screened  the  deficiencies  for  reported 
disposition  actions  that  resulted  in  tested  products  being  returned  to  stock,  used 
without  modification,  or  where  no  action  on  the  tested  product  was  deemed 
necessary.  A  total  of  231  of  the  671  reported  quality  deficiencies  met  these 
cause  and  disposition  characteristics.  For  the  231  deficiencies,  we  appraised  the 
consistency  of  the  quality  assurance  specialists’  evaluation  process  to  determine 
the  cause  of  and  resulting  disposition  action  of  the  quality  deficiency.  Table  6 
displays  our  analysis  of  the  FY  1999  finalized  test  results. 


Table  6.  Analysis  of  Finalized  Test  Results  -  FY  1999 


Total 

Percentage  of 

Test  Results 

Test  Results 

Test  Failures 

Test  Failures 

Finalized  in 

Reviewed 

Properly 

Not  Properly 

FY  1999 

Processed 

Processed 

Columbus 

553 

154 

105 

32 

Richmond 

101 

60 

44 

34 

Philadelphia 

17 

17 

17 

0 

Total 

671 

231 

166 

36 

Defense  Supply  Center,  Columbus.  Quality  assurance  specialists  properly 
identified  actual  causes  and  provided  appropriate  disposition  actions  for  105  of 
154  deficiency  reports.  The  105  deficiency  reports  included  72  reports  that 
properly  identified  contractor  noncompliance  as  the  cause  of  the  deficiency. 

The  remaining  33  deficiency  reports  properly  identified  causes  supporting  a 
simplified  investigation  or  no  investigation.  Quality  assurance  specialists 
improperly  coded  test  results  for  49  of  154  deficiency  reports  when  the  results 
revealed  that  the  contractor  caused  the  test  failures.  Thirteen  of  the  49  test 
failures  were  reported  as  major  deficiencies.  Twenty  of  the  49  test  failures 
involved  critical  items.  We  interviewed  13  quality  assurance  specialists  that 
reviewed  34  of  the  49  deficiency  reports  to  determine  their  rationale  for  not 
coding  the  cause  of  the  deficiency  report  as  contractor  noncompliance.  The 
quality  assurance  specialists  provided  the  following  information  regarding  the 
test  results. 

•  The  quality  assurance  specialist  could  not  identify  the  contractor  or 
contract  number  of  the  tested  product;  thus,  the  specialist  contended  that 
no  remedy  was  available  from  the  contractor. 
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•  The  defect  was  considered  minor  (although  the  reported  deficiency  was 
not  cosmetic  in  nature) . 

•  No  previous  complaints  were  raised  against  the  contractor  or  the  tested 
product. 

•  Low-dollar  value  or  age  of  the  tested  product. 

•  Disagreement  with  the  test  results  or  with  the  product  specifications. 

The  quality  assurance  specialists  applied  inconsistent  criteria  when  the  actual 
causes  of  the  deficiencies  were  disclosed.  The  specialists  did  not  take  any 
corrective  action  or  provide  any  test  result  notifications  to  the  supply  points  for 
22  of  the  49  improperly  coded  deficiency  reports.  Twelve  of  the  22  products 
that  had  no  corrective  action  involved  critical  products,  and  3  of  the  12  critical 
products  encountered  major  test  failures.  These  major  test  failures  involved 
avionics  components  for  fighter  and  cargo  aircraft,  helicopters,  and 
microcircuits  for  fighter  aircraft.  Products  with  the  same  or  similar  defects  may 
be  in  the  supply  system  because  notifications  were  not  circulated.  We  notified 
the  PVP  manager  of  major  test  failures  encountered  on  critical  products  in  an 
effort  to  facilitate  the  corrective  action  process.  Uniform  investigative 
procedures  should  be  addressed  in  a  standard  training  program. 

Defense  Supply  Center,  Richmond.  Quality  assurance  specialists  properly 
identified  actual  causes  and  provided  appropriate  disposition  actions  for  26  of 
60  deficiency  reports.  Twenty  deficiency  reports  properly  identified  contractor 
noncompliance  as  the  cause  of  the  deficiency.  The  remaining  six  deficiency 
reports  properly  identified  causes  that  supported  simplified  investigations  or  no 
investigations.  Quality  assurance  specialists  improperly  coded  test  failures  for 
19  of  60  deficiency  reports  when  test  results  revealed  that  the  contractor  was  at 
fault.  The  remaining  15  test  failures  were  dismissed  without  investigation.  Ten 
of  the  34  test  failures  were  classified  as  major  deficiencies,  17  of  the  34  test 
failures  involved  critical  items.  We  interviewed  6  quality  assurance  specialists 
that  reviewed  the  34  deficiency  reports  to  determine  their  rationale  for  not 
coding  the  cause  of  the  deficiency  report  as  contractor  noncompliance.  The 
quality  assurance  specialists  provided  the  following  information  regarding  the 
test  results. 

•  One  quality  assurance  specialist  summarily  dismissed  15  deficiency 
reports  contending  that  the  age  of  the  tested  products  and  the  lack  of 
investigative  personnel  precluded  further  action. 

•  A  possible  inventory  shortage  of  the  test-failed  product  would  result 
from  suspending  it  from  subsequent  issue. 

•  The  contracting  office  disagreed  that  the  contractor  was  at  fault  and 
changed  the  cause  of  the  deficiency. 

•  The  quality  assurance  specialist  did  not  wish  to  “punish”  the  contractor 
for  minor  defects  disclosed  in  product  testing. 
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Again,  quality  assurance  specialists  applied  inconsistent  criteria  when  the  actual 
causes  of  the  deficiencies  were  disclosed.  The  specialists  did  not  take  corrective 
action  or  provide  notifications  to  supply  points  for  22  of  34  finalized  deficiency 
reports.  Fifteen  of  the  22  products  that  had  no  corrective  action  were  critical 
and  7  of  the  15  critical  products  had  major  test  failures.  These  major  test 
failures  involved  fighter  and  cargo  aircraft,  helicopters,  radar  equipment  and 
tank  parts.  Inventory  integrity  may  have  been  compromised  by  the  lack  of 
corrective  actions  taken  on  major  failures  involving  critical  products.  We 
notified  the  PVP  manager  of  major  test  failures  encountered  on  critical  products 
in  an  effort  to  facilitate  corrective  actions. 

Defense  Supply  Center,  Philadelphia.  Automated  records  from  the 
Philadelphia  center  indicated  17  finalized  FY  1999  quality  deficiency  reports. 
The  reports  properly  identified  the  cause,  and  disposition  actions  were 
commensurate  with  the  decisions  rendered  in  the  tested  product.  However,  we 
discovered  a  management  control  weakness  involving  reporting  test  result 
failures  to  the  automated  database  for  subsequent  investigation  and  dissolution. 
We  identified  22  test  failures  representing  over  38,000  components  for  F-15 
fighter  jets,  C-130  cargo  aircraft,  and  the  Navy  DDG  class  destroyer.  We 
addressed  our  concerns  to  the  Commander,  Defense  Supply  Center, 
Philadelphia,  on  June  8,  2000.  The  commander  took  immediate  corrective 
action  and  provided  detailed  written  responses  on  July  19,  2000.  See  Appendix 
B  for  the  inventory  integrity  issue,  and  Appendix  C  for  the  corrective  action 
responses. 

Impact  of  Deficiency  Report  Codes 


Past  Performance.  A  contractor’s  past  performance  information  was  collected 
and  translated  into  a  numeric  score  using  the  automated  best  value  computerized 
system.  The  contracting  officer  used  this  score  in  the  evaluation  process  when 
awarding  contracts.  Contractors  received  performance  scores  for  each  Federal 
supply  class  procurement  and  may  have  multiple  scores.  However,  contractors 
had  one  Defense  Logistics  Agency  score,  which  compiled  the  contractor’s 
Federal  supply  score  for  all  business  conducted  with  the  agency.  The  quality 
was  reduced  when  deficiency  report  results  were  coded  as  contractor 
noncompliance. 

Contractor  Quality  Scores.  We  evaluated  the  impact  on  the  Automated  Best 
Value  System  for  60  of  83  deficiency  reports  that  should  have  been  coded  as 
contractor  noncompliance.  The  60  deficiency  reports  affected  the  overall  scores 
of  54  contractors.  The  Defense  Supply  Center,  Columbus,  scoring  range  should 
have  been  reduced  between  .05  percent  and  20  percent  for  the  average  Federal 
supply  class  score.  The  Defense  Supply  Center,  Richmond,  scoring  range 
should  have  been  reduced  between  .05  percent  and  13.3  percent  for  the  average 
Federal  supply  class  score.  Conversely,  we  confirmed  that  contractor  quality 
scores  were  properly  reduced  when  the  contractor  was  rated  noncompliant  on 
the  deficiency  report.  When  the  deficiency  reports  were  improperly  coded  as 
contractor  compliance,  scores  for  quality  were  overstated,  did  not  reflect  the 
actual  contractor  performance,  and  earned  the  contractor  an  advantage  on  future 
contract  awards. 
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Nonconforming  Product  Implications.  Financial  and  inventory  quality 
implications  were  impacted  when  investigations  and  corrective  actions  were  not 
completed  on  nonconforming  products.  Although  test  results  disclosed 
deficiencies,  little,  if  anything  was  done  to  ensure  that  deficient  items  were  not 
in  inventory  or  issued  to  customers.  As  a  minimum,  items  from  the  same  lots 
as  the  deficient  items  should  be  reviewed.  Table  7  displays  the  potential 
inventory  and  financial  impact  of  the  587  quality  deficiency  reports  that  were 
closed  in  FY  1999.  We  identified  at  least  362  contractors  that  provided  DoD 
items  reported  as  deficient  on  the  587  quality  deficiency  reports.  These  vendors 
provided  at  least  456,000  items,  valued  at  over  $3.9  million,  between  1998  and 
2000. 


Table  7. 

Inventory  Impact  of  FY  1999  Deficiency  Reports  on  Quality  Issues 

Supply 

Center 

FY  1999 

Deficiency  Reports 

Identified 

Products 

Identified 

Contractors 

Products 
Procured  1998 
to  2000 

Product  Value 

Columbus 

469 

421 

267 

403,236 

$3,364,934 

Richmond 

101 

93 

90 

42,277 

$545,819 

Philadelphia 

17 

17 

5 

10,715 

$47,078 

Total 

587 

531 

362 

456,228 

$3,957,831 

We  recognize  that  not  all  the  items  purchased  from  the  362  contractors  were 
deficient.  However,  we  also  believe  that  when  a  contractor  produced  one 
deficient  item,  others  may  follow,  which  increased  the  risks  substantially.  By 
not  citing  the  contractor  as  noncompliant,  financial  remedy  was  lost.  Inventory 
quality  was  also  diminished  when  corrective  actions  were  not  taken  on  products 
that  ultimately  were  issued  and  not  fully  operational.  Major  quality  deficiencies 
should  trigger  the  need  for  additional  product  tests  from  a  contractor’s  products 
to  determine  whether  the  deficiency  was  pervasive  to  the  remaining  inventory. 
Finally,  additional  testing  of  other  products  purchased  from  a  contractor  that 
provided  nonconforming  products  may  also  be  warranted. 

Quality  Assurance  Management 


The  deficiency  report  inconsistencies  were  not  uniformly  distributed  at  all 
supply  centers.  We  analyzed  inconsistent  deficiency  reports  at  the  Defense 
Supply  Centers,  Columbus  and  Richmond.  Forty-five  of  49  (92  percent) 
deficiency  reports  were  concentrated  in  2  of  the  4  operating  elements  at  the 
Defense  Supply  Center,  Columbus.  Twenty-five  of  34  (74  percent)  of  the 
deficiency  reports  were  concentrated  at  3  of  the  1 1  operating  elements  at  the 
Defense  Supply  Center,  Richmond.  Quality  assurance  specialists  and 
supervisory  reviews  were  not  evident  during  the  evaluation  and  final 
adjudication  of  deficiency  reports  at  these  supply  centers.  Quality  assurance 
management  at  the  Defense  Supply  Center,  Columbus,  is  planning  to  institute 
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procedural  instructions  for  supervisors  to  administer  review  and  oversight 
procedures.  A  DLA  training  program  would  uniformly  strengthen  the  decision 
making  process  for  all  personnel. 

Recommendations  and  Management  Comments 

A.  We  recommend  that  the  Director,  Defense  Logistics  Agency: 

1.  Develop  and  implement  quality  assurance  specialist  training  and 
supervisory  review  procedures  at  Defense  Supply  Centers  to  uniformly 
require  the  quality  assurance  specialist  to: 

a.  Determine  whether  the  contractor  was  at  fault  as  a  result  of  a 
test  failure  generated  from  a  product  verification  program  test. 

b.  Document  the  causative  factors  for  product  test  failures. 

c.  Code  all  contractor  noncompliance  performance  test  failures. 

d.  Document  disposition  and  corrective  actions  for  failed  product 
inventories  with  failed  test  results. 

2.  Using  FY  1999  test  results,  determine  whether  nonconforming 
products  purchased  from  contractors  and  stocked  in  depots 
should  be  suspended  or  reevaluated. 

Management  Comments.  The  Defense  Logistics  Agency  concurred  and  stated 
that  implementation  of  current  Defense  Logistics  Agency  Directives  and 
Instructions  will  address  all  areas  listed  as  deficient  in  this  report.  In  addition, 
the  agency  stated  that  certification  procedures  outlined  in  the  Quality  Assurance 
Technical  Development  Program  and  through  on-the-job  training  will  properly 
address  deficiencies  in  quality  deficiency  reporting.  The  Defense  Logistics 
Agency  stated  that  it  has  begun  to  reevaluate  and  take  appropriate  corrective 
action  on  the  FY  1999  test  results  identified  in  this  report. 


15 


Appendix  A.  Audit  Process 


Scope 


Work  Performed.  For  our  analysis  of  product  selection  for  quality  testing,  we 
obtained  a  listing  of  4,705  FY  1999  randomly  selected  products  from  the 
Product  Verification  Program  offices.  We  analyzed  the  criticality  for  3,210  of 
the  4,705  randomly  selected  products  and  costs  for  1,063  nonrandomly  selected 
products  to  determine  whether  the  products  met  the  criteria  in  DLA  Directive 
4105.20,  “Product  Verification  Program  for  Inventory  Control  Points, "January 
1995.  We  analyzed  245  nonrandomly  selected  products  to  determine  whether 
the  supply  centers  selected  products  in  accordance  with  management  initiatives. 

For  our  analysis  of  the  use  of  test  results,  we  obtained  and  evaluated 
231  product  test  failures  that  were  documented  by  the  program.  The  supply 
centers  compiled  a  listing  of  1,163  FY  1999  laboratory  test  failures.  We  limited 
our  analysis  to  231  test  failures  generated  from  program  testing  that  involved 
quality  discrepancies  with  corresponding  disposition  action  that  resulted  in  the 
tested  product  being  returned  to  stock,  used  without  modification,  or  no  further 
action  was  taken.  We  subsequently  analyzed  whether  quality  assurance 
specialists  properly  rated  the  test  failures,  their  severity,  and  whether  the  test 
result  warranted  corrective  action.  We  also  measured  the  impact  of  improperly 
evaluated  test  results  on  the  Automated  Best  Value  System  that  measures  a 
contractor’s  quality  performance.  We  measured  60  deficiency  reports  affecting 
the  system  scores  of  54  contractors. 

DoD-Wide  Corporate  Level  Government  Performance  and  Results  Act 
(GPRA)  Goals.  In  response  to  GPRA,  the  Secretary  of  Defense  annually 
establishes  DoD-wide  corporate  level  goals,  subordinate  performance  goals,  and 
performance  measures.  There  is  no  DoD-wide  goal  that  specifically  addressed 
quality  of  DoD  material  assets. 

General  Accounting  Office  High-Risk  Area.  The  General  Accounting  Office 
has  identified  several  high-risk  areas  in  the  DoD.  This  report  provides  coverage 
of  the  Defense  Inventory  Management  high-risk  area. 

Methodology 


Use  of  Computer-Processed  Data.  We  relied  on  computer-processed  data 
from  the  DLA  Standard  Automatic  Material  Management  System.  From  this 
automated  system  and  its  subsystems,  we  obtained  universes  of  random  and 
nonrandom  selected  products  for  testing.  We  relied  on  the  Defense  Logistics 
Agency  Customer  Depot  Complaint  System  for  customer  and  PVP-generated 
Product  Quality  Deficiency  Reports.  We  relied  on  the  Automated  Best  Value 
System  to  obtain  contractor  quality  scores.  Although  we  did  not  perform  a 
formal  reliability  assessment  of  the  computer-processed  data,  the  information 
obtained  generally  agreed  with  the  information  in  the  computer-processed  data 
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We  did  not  find  errors  that  would  preclude  use  of  the  computer-based  data  to 
meet  the  audit  objectives  or  that  would  change  the  conclusions  in  this  report. 


Audit  Type,  Dates,  and  Standards.  We  performed  this  program  audit  from 
March  to  October  2000  in  accordance  with  auditing  standards  issued  by  the 
Comptroller  General  of  the  United  States,  as  implemented  by  the  Inspector 
General,  DoD.  We  included  tests  of  management  controls  considered 
necessary. 

Contacts  During  the  Audit.  We  visited  or  contacted  individuals  within  DoD. 
Further  details  are  available  upon  request. 

Management  Control  Program  Review 


DoD  Directive  5010.38,  “Management  Control  (MC)  Program,”  August  26, 
1996,  and  DoD  Instruction  5010.40,  “Management  Control  (MC)  Program 
Procedures,”  August  28,  1996,  require  DoD  organizations  to  implement  a 
comprehensive  system  of  management  controls  that  provides  reasonable 
assurance  that  programs  are  operating  as  intended  and  to  evaluate  the  adequacy 
of  the  controls. 

Scope  of  the  Review  of  the  Management  Control  Program.  In  IG,  DoD, 
Report  No.  D200 1-002,  “Defense  Logistics  Agency  Customer  Return 
Improvement  Initiative,”  October  12,  2000,  we  reviewed  the  adequacy  of  the 
Product  Verification  Program  management  controls  over  the  screening  of 
potentially  nonconforming  products  received  through  customer  returns.  We 
reviewed  the  accuracy  and  the  extent  that  the  Customer  Return  Improvement 
Initiative  exercised  in  notifying  depots  of  potentially  defective  products.  During 
this  audit,  we  reviewed  the  adequacy  of  the  management  controls  over  the  use  of 
test  results  derived  from  Product  Verification  Program  testing.  Specifically,  we 
appraised  management  controls,  including  supervisory  review  and  oversight, 
that  would  consistently  identify  and  record  the  source,  disposition  and  possible 
correction  of  products  that  failed  quality  testing.  We  did  not  review 
management’s  self-evaluation  applicable  to  these  controls. 

Adequacy  of  Management  Controls.  We  identified  material  management 
control  weaknesses  during  the  review  of  the  Customer  Returns  Improvement 
Initiative  Program  and  in  the  use  of  test  results  derived  from  Product 
Verification  Program  testing,  as  defined  by  DoD  Instruction  5010.40. 
Management  controls  for  the  Customer  Returns  Improvement  Program  were  not 
adequate  to  ensure  that  potentially  defective  products  received  through  customer 
returns  were  made  available  for  issue  at  some  DLA  depots.  Likewise, 
management  controls  for  the  use  of  test  results  derived  from  Product 
Verification  Program  testing  were  not  adequate  to  ensure  that  consistent  and 
effective  test  results  data  was  recorded  and  subsequent  corrective  action  were 
taken  to  prevent  potentially  nonconforming  products  from  becoming  available 
for  issue.  Recommendations  1  and  2  of  the  Customer  Returns  Improvement 
Initiative  Program  report,  and  Recommendations  B.l.a.,  B.l.b.,  B.l.c.  and 
B.l.d.  of  this  report,  if  implemented,  will  improve  the  overall  integrity  of  DLA 
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managed  products.  A  copy  of  both  reports  will  be  provided  to  the  senior 
official  responsible  for  management  controls  in  the  Defense  Logistics  Agency. 


Prior  Coverage 

Inspector  General,  DoD,  Report  No.  D-200 1-002,  “Defense  Logistics  Agency 
Customer  Returns  Improvement  Initiative  Program,”  October  12,  2000. 
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Appendix  B.  Test  Results  -  Defense  Supply 

Center,  Philadelphia  (Alert 
Memorandum) 


INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRIVE 
ARLINGTON,  VIRGINIA  22202-2885 


JUN  8  2000 


MEMORANDUM  FOR  COMMANDER,  DEFENSE  SUPPLY  CENTER  PHILADELPHIA 
SUBJECT:  Inventory  Integrity  -  Nonconforming  Items 


As  part  of  our  audit  of  the  Defense  Logistics  Agency  Product  Verification  Program,  we 
observed  an  ongoing  condition  that  warrants  your  immediate  attention.  At  the  Defense  Supply 
Center,  Philadelphia,  Pennsylvania,  we  noted  an  internal  management  control  weakness 
affecting  quality  assurance  procedures  within  the  General  and  Industrial  Directorate. 
Specifically,  we  found  that  all  test  failures  under  the  Product  Verification  Program  involving 
consumable  items  managed  by  the  General  and  Industrial  Directorate  were  not  reported  in  the 
automated  logistical  database  nor  subsequently  investigated  or  disposed  of  appropriately.  As  a 
result,  nonconforming  items  (test  failures)  may  have  been  issued  to  customers  from  current 
inventory  stock. 

We  requested  that  the  Test  and  Evaluation  Office,  Logistics  Support  Office,  General 
and  Industrial  Directorate  provide  us  with  a  list  of  nonconforming  items  identified  during 
testing  in  Fiscal  Year  1999  under  the  Product  Verification  Program.  The  Test  and  Evaluation 
Office  identified  22  nonconforming  (test  failed)  items,  which  represented  more  than  38,000 
units  purchased.  The  Office  further  indicated  that  these  nonconforming  items  contained  13 
major  defects  and  9  minor  defects.  A  major  defect  is  considered  likely  to  result  in  a  failure  or 
in  a  reduction  in  the  usability  of  the  item  for  its  intended  purpose.  Furthermore,  9  of  the  13 
major  defects  involved  critical  items.  These  critical  items  relate  to  weapon  systems  such  as  the 
F-15  and  Harrier  fighter  jets,  C-130  cargo  aircraft,  and  the  Navy  DDG  class  destroyers. 

Before  the  General  and  Industrial  Directorate  reorganized  in  July,  1999,  quality 
assurance  personnel  in  the  Directorate  were  responsible  for  entering  nonconforming  items 
identified  through  testing  into  the  Customer  Depot  Complaint  System.  These  personnel  also 
distributed  the  test  results  to  the  appropriate  Quality  Assurance  Specialists  for  investigation  and 
resolution.  Quality  Assurance  Specialists  investigated  test  validity,  contractor  responsibility 
for  the  nonconforming  items,  parts  usability,  and  the  quantity  and  location  of  the  remaining 
items.  Subsequently,  Quality  Assurance  Specialists  may  determine  that  disposal  of  the  test 
sample  and  remaining  inventory  was  necessary  to  protect  the  integrity  of  the  inventory. 

We  determined  that  the  22  items  that  failed  testing  in  Fiscal  Year  1999  were  not  entered 
into  the  Customer  Depot  Complaint  System  and  that  test  results  were  not  distributed  to  the 
appropriate  Quality  Assurance  Specialists  for  investigation  and  resolution.  As  much  as  18 
months  have  passed  since  failed  item  were  identified.  Consequently,  nonconforming  items, 
most  likely,  have  been  issued.  The  crucial  functions  of  entering  test  data  into  the  Customer 
Depot  Complaint  System  and  distributing  it  to  the  Quality  Assurance  Specialists  have  not  been 
performed  since  the  General  and  Industrial  Directorate  has  been  reorganized. 
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We  recommend  that  the  General  and  Industrial  Directorate  immediately  perform  the 
functions  related  to  the  recording  and  notification  of  test  failures.  We  also  recommend  that  the 
Directorate  address  the  22  items  that  failed  testing  in  Fiscal  Year  1999  by  notifying  all 
respective  Quality  Assurance  Specialists  of  the  nonconforming  items  and  insuring  that  all 
customers  are  properly  informed.  Without  this  crucial  link  between  product  testing  and  quality 
assurance,  the  likelihood  of  issuing  nonconforming  items  containing  major  defects  has 
continued  and  will  continue.  We  consider  this  condition  to  be  an  internal  management  control 
weakness  and  recommend  that  it  receive  immediate  attention. 

Please  response  to  any  actions  taken  or  contemplated  by  July  5,  2000.  If  you  have  any 
questions  on  this  matter,  contact  Mr.  Nick  Como  at  (703)  604-9215. 


Director 

Contract  Management  Directorate 
cc: 

DLA  Product  Verification  Program  Manager 
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Appendix  C.  Test  Results  -  Defense  Supply 

Center,  Philadelphia  (Command 
Reply) 


N  REPLY 
REFER  TO 


DEFENSE  LOGISTICS  AGENCY 
DEFENSE  SUPPLY  CENTER  PHILADELPHIA 
700  ROBBINS  AVENUE 

PHILADELPHIA,  PENNSYLVANIA  19111-5092 


D 


JULY  19,  2000 


MEMORANDUM  FOR  DIRECTOR,  CONTRACT  MANAGEMENT  DIRECTORATE 
DOD  INSPECTOR  GENERAL 

SUBJECT:  Inventory  Integrity  -  Final  Report  on  Nonconforming  Items 


As  a  follow  up  to  your  June  8,  2000  Memorandum  on  the  subject  and  our  June  12,  2000 
interim  response,  this  correspondence  summarizes  the  actions  we  have  undertaken  to 
rectify  your  Team’s  findings  and  to  ensure  effective,  systemic  controls  of  our  quality 
assurance  process. 

Written  notification  of  your  Team’s  specific  findings  were  provided  to  the  Quality 
Assurance  Specialists  (QASs)  in  our  buying  units.  Included  with  that  notification  were 
the  related  test  project  folders  identifying  all  non-conformance  test  findings.  The  QASs 
are  currently  investigating  the  non-conformance  findings  and  are  pursuing  appropriate 
follow-up  actions,  as  warranted,  to  rectify  the  situation-i.e.,  advising  customers,  screening 
the  inventory,  and  performing  other  related  tasks  to  effect  responsible  project  disposition. 

Written  instructions  were  issued  to  all  Test  and  Evaluation  (T&E)  personnel  reiterating 
procedures  to  be  followed  for  proper  processing  of  test  findings.  In  addition,  detailed 
standard  operating  procedures  addressing  that  process  were  developed  and  provided  to  all 
General  and  Industrial  Directorate  T&E  and  QAS  personnel  and  their  supervisory  chain 
to  the  Office  Chief  level.  These  procedures  stipulate  the  requirement  and  process  for 
entry  into  the  Customer  Depot  Complaint  System  of  non-conformance  findings  resulting 
from  random  inventory  testing  projects  and  provide  the  needed  oversight  to  ensure  this 
action  is  systematically  effected. 


5ANIEL  g.  MONGEON 
Brigadier  General,  USA 
Commanding  / 


DLA  Product  Verification  Program  Manager 

RADM  Stone,  SC,  USN,  (J-3),  Commander,  Defense  Logistics  Agency 


21 


Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
Deputy  Under  Secretary  of  Defense  (Logistics) 

Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Financial  Management  and  Comptroller) 

Auditor  General,  Department  of  the  Army 

Department  of  the  Navy 

Naval  Inspector  General 

Auditor  General,  Department  of  the  Navy 

Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 

Auditor  General,  Department  of  the  Air  Force 

Other  Defense  Organizations 

Director,  Defense  Logistics  Agency 

Director,  Product  Verification  Program 
Director,  Defense  Supply  Center  Columbus 
Director,  Defense  Supply  Center  Philadelphia 
Director,  Defense  Supply  Center  Richmond 

Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget 
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Congressional  Committees  and  Subcommittees,  Chairman  and 
Ranking  Minority  Member 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Reform 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 
Committee  on  Government  Reform 

House  Subcommittee  on  National  Security,  Veterans  Affairs,  and  International 
Relations,  Committee  on  Government  Reform 
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Defense  Logistics  Agency  Comments 


DEFENSE  LOGISTICS  AGENCY 
HEADOUARTERS 

8725  JOHN  J.  KINGMAN  ROAD.  SUITE  2533 
FT.  BELVOIR.  VIRGINIA  22060-6221 


IN  REPLY 
REFER  TO 


FEB  "6  2001 


MEMORANDUM  FOR  DoD  IG 

SUBJECT:  Audit  Report  on  the  Defense  Logistics  Agency  Product  Verification 
Program  (Project  No.  D20000CF-01 1 1 .001) 


In  response  to  your  memorandum  dated  November  15, 2000,  subject  as  above,  wc 
have  reviewed  the  draft  audit  report,  and  our  comments  are  attached.  Our  point  of 
contact  is  Mr.  Michael  Shields,  J-3342,  (703)  767-2629  (michael_shields@hq.dla.mil). 


Attachment 


D.  H.  Stone 

Rear  Admiral,  SC,  USN 
Director 

Logistics  Operations 


Federal  ReeveUno  Program  1 


i  Printed  on  Recycled  Paper 
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Subject:  Defense  Logistics  Agency  Product  Verification  Program 
(Project  No.  D2000CF-01 1 1.001) 

Finding  A:  Funds  for  product  testing  were  not  used  in  the  most  efficient  manner  and 
DoD  had  no  assurance  that  critical  products  would  perform  as  expected.  (See  page  2  of 
report). 

DLA  Comments:  Non-concur.  DLA  currently  uses  a  Sampling  Assistance  Model 
developed  by  the  DLA  Office  of  Research  and  Resource  Analysis  (DORRA)  and 
previously  approved  by  the  DoDIG.  This  model  generates  the  sample  population  for 
random  testing  from  all  available  ready  for  issue,  condition  code  A  wholesale  inventory. 
The  objective  of  this  random  sampling  is  to  provide  a  statistical  inference  of  the  quality 
level  of  all  ready  to  issue  stock.  The  weighting  of  criticality  would  skew  the  inference 
and  result  in  an  undcsired  metric.  Critical  items  arc  evaluated  by  Product  Verification 
Program  nonrandom  or  directed  testing  projects. 

Finding  B:  For  two  of  the  three  Defense  Supply  Centers,  test  failures  were  not 
consistently  investigated  and  required  actions  on  test  failures  were  not  always  taken. 
Inconsistent  adjudication  and  ratings  of  test  results  hindered  the  two  Defense  Supply 
Centers  from  resolving  contractor  issues  for  36  percent  of  the  23 1  FY  1999  tests 
reviewed,  inflated  quality  ratings  for  as  many  as  54  contractors  and  allowed  potentially 
nonconforming  products  to  remain  available  for  issue.  (See  page  9  of  report). 

DLA  Comments:  Concur.  The  Primary  Level  Field  Activities  (PLFAs)  will  review 
current  Product  Quality  Deficiency  Report  (PQDR)  processes  to  ensure  complete 
investigations  and  satisfactory  actions  are  taken  to  correct  existing  deficiencies. 

Recommendations 

A.  Director,  Defense  Logistics  Agency: 

1.  Establish  random  testing  selection  methods  in  Defense  Supply  Center 
product  verification  programs  to: 

a.  Sample  all  stored  depot  products. 

b.  Factor  product  criticality. 

c.  Include  direct  vendor  delivery  procurements. 
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DLA  Comments:  Non-concur.  DLA  is  currently  developing  a  new  program  to  replace 
the  random  sampling  portion  of  the  PVP.  This  program  uses  a  total  quality  cost 
methodology  to  direct  nonrandom  testing  selection  to  high  cost  drivers  or  factors  for 
example,  depot  location,  item  criticality,  delivery  method,  and  customer  complaint 
trends,  which  represent  the  highest  potential  rates  of  return  in  terms  of  customer 
satisfaction  and  reduced  internal  operational  costs.  The  FY  2000-2005  Performance 
Contract  between  the  Defense  Management  Council  specifies  this  new  program  as  a 
replacement  for  the  current  random  testing  Product  Conformance  metric  in  FY  02. 

2.  Implement  nonrandom  testing  initiatives  prescribed  in  Defense  Logistics 
Agency  Directive  4105.20. 

DLA  Comments:  Concur.  Our  Centers  are  currently  performing  nonrandom  testing  as 
stated  in  the  report.  We  are  also  implementing  a  new  program  as  described  above.  The 
model  will  change  our  focus  from  testing  just  categories  of  items  to  testing  particular 
items,  locations,  and  suppliers  based  on  potential  risk  and  past  performance. 

3.  Prepare  trend  analyses  from  quality  deficiency  reports  to  identify 

nonconforming  inventory  for  the  following  products: 

a.  Critical. 

b.  Highly  requisitioned. 

c.  High-cost,  high-dollar,  or  high-volume  products. 

d.  Items  with  known  or  potential  deficiencies. 

e.  Items  from  contractors  with  poor  performance  histories. 

DLA  Comments:  Concur.  DLAI  4155.2  requires  DLA  ICP  PQDR  control  points  to 
perform  trend  analysis  on  both  customer  PQDRs  and  deficiencies  identified  through  the 
PVP.DLA  will  direct  ICP  Commanders  to  implement  DLA  policy  requirements  as 
defined  by  this  instruction.  In  addition,  the  Total  Quality  Cost  Model  will  also  identify 
trends  of  repetitive  deficiencies  or  other  high  risk  cost  drivers  such  as  those  cited  above 
for  corrective  action  to  improve  customer  satisfaction  and  drive  down  operational  costs. 

B.  Director,  Defense  Logistics  Agency: 

1.  Develop  and  implement  quality  assurance  specialist  training  and 
supervisory  review  procedures  at  Defense  Supply  Centers  to  uniformly 
required  the  quality  assurance  specialist  to: 

a.  Determine  whether  the  contractor  was  at  fault  as  a  result  of  a  test 

failure  generated  from  a  product  verification  program  test. 

b.  Document  the  causative  factors  for  product  test  failures 

c.  Code  all  contractor  noncompliance  performance  test  failures. 

d.  Document  disposition  and  corrective  actions  for  failed  product 

inventories  with  failed  test  results. 
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DLA  Comments:  Concur.  DLA  Directive  4155.24  provides  general  policy  and  DLA 
Instruction  4155.24  “  Product  Quality  Deficiency  Reporting  Program”  provides  detailed 
procedures  for  reporting  and  processing  PQDRs.  Additional  information  on  the 
processing  of  PQDRs  is  also  available  in  DLA  Directive  4155.2  and  DLA  Instruction 
4155.2  “DLA  Quality  Assurance  Program  for  DLA  Supply  Centers.”  Quality  Assurance 
Specialists  (QASs)  also  require  certification  in  Center  QA  Systems  under  the  Quality 
Assurance  Technical  Development  Program.  This  certification  addresses  all  areas  cited 
above.  In  addition  to  formal  training,  each  PLFA  will  provide  on  the  job  and  periodic 
training  to  personnel  processing  and  reviewing  PQDRs. 

2.  Using  FY 1999  test  results,  determine  whether  nonconforming  products 
purchased  from  contractors  and  stocked  in  depots  should  be  suspended  or 
reevaluated. 

DLA  Comments:  Concur.  DSCP  has  already  taken  action  to  review  FY  99  test  results 
and  take  all  appropriate  follow  up  actions  including  those  cited  in  the  recommendation. 
These  actions  were  reported  in  the  DSCP  Command  Reply  on  July  19, 2000,  attached  as 
appendix  C  of  this  report.  DSCR  is  currently  re-evaluating  their  FY  99  results  and  will 
take  appropriate  action  based  on  the  results  of  this  re-evaluation.  DSCC  requests  the 
DoDIG  identify  the  49  nonconforming  test  results  reported  on  Table  6,  page  10  of  the 
report.  DSCC  will  re-evaluate  FY  99  results  once  the  49  have  been  identified  and  will 
take  appropriate  action  based  on  the  results  of  this  re-evaluation. 


Audit  Team  Members 

The  Contract  Management  Directorate,  Office  of  the  Assistant  Inspector  General  for 
Auditing,  DoD,  prepared  this  report.  Personnel  of  the  Office  of  the  Inspector  General, 
DoD,  who  contributed  to  the  report  are  listed  below. 

Paul  J.  Granetto 
Terry  L.  McKinney 
Nicholas  E.  Como 
Lusk  F.  Penn 
Samuel  J.  Scumaci 
Chuck  J.  Chin 
Cecil  B.  Tucker 
Phillip  R.  Sartori 


